IBM Books

Software User's Guide Version 3.4


Using the Event Logging System (ELS)

This chapter describes the Event Logging System (ELS). The ELS continually logs all events, filtering them according to parameters that you select. A combination of operational counters and the ELS provides information for monitoring the health and activity of the system. The information is divided into the following sections:


What is ELS?

ELS is a monitoring system and an integral part of the device operating system. ELS manages the messages logged as a result of device activity. Use ELS commands to set up a configuration that sorts out only those messages you feel are important. You can then display the messages on the console terminal screen, log them to a remote workstation, or send the messages to a network management station using Simple Network Management Protocol (SNMP) traps.

The ELS system and the operational counters are the best troubleshooting tools you have to isolate problems in the device. A quick scan of the event messages will tell you whether the device has a problem and where to start looking for it.

In the ELS configuration environment, the commands are used to establish a default configuration. This default configuration does not take effect until the device reinitializes.

Occasionally, it is helpful to temporarily view messages using parameters other than was set up in the ELS configuration environment, without having to reinitialize the device. The ELS operating and monitoring environment is used to:

Note:Specific ELS messages are described in the IBM Event Logging System Messages Guide.

ELS is a subprocess that you access from the OPCON process.


Entering and Exiting the ELS Configuration Environment

The ELS configuration environment (available from the CONFIG process) is characterized by the ELS Config> prompt. Commands entered at this prompt create the ELS default state that takes effect after you restart the device. These commands are described in greater detail later in this chapter.

Configuration commands that have subsystem, group, or event as a parameter are executed in the following order:

To set a basic ELS configuration, enter the display subsystem all standard command at the ELS Config> prompt. This command configures the ELS to display messages from all subsystems with the STANDARD logging level (that is, all errors and unusual informational comments).
Note:The device does not have a default ELS configuration. You must enter the ELS configuration environment and set the default state.

To enter the ELS configuration environment from OPCON:

  1. Enter the configuration command. The console displays the CONFIG prompt (Config>). If the prompt does not appear when you first enter CONFIG, press enter.
  2. At the CONFIG prompt, enter the following command to access ELS:
    Config> eve
    

    The console displays the ELS configuration prompt (ELS config>). Now, you can enter ELS configuration commands.

To leave the ELS configuration environment, enter the exit command.


Event Logging Concepts

This section describes how events are logged and how to interpret messages. Also described are the concepts of subsystem, event number, and logging level. A large part of ELS function is based on commands that accept the subsystem, event number, and logging level as parameters.

Causes of Events

Events occur continuously while the device is operating. They can be caused by any of the following reasons:

When an event occurs, ELS receives data from the system that identifies the source and nature of the event. Then ELS generates a message that uses the data received as part of the message.

Interpreting a Message

This section describes how to interpret a message generated by ELS. Figure 4 shows the message contents.

Figure 4. Message Generated by an Event

amo4a008

The information illustrated in Figure 4 as well as the ELS logging level information displayed with the list subsystem command is as follows:

Subsystem

Subsystem is a predefined short name for a device component, such as a protocol or interface. In Figure 4, GW identifies the subsystem through which this event occurred.

Other examples of subsystems include IP and ETH. On a particular device, the actual subsystems present depend on the hardware and software configured for that device. You can use the list subsystem command described in this chapter to see a list of the subsystems on your device.

Enter the subsystem as a parameter to an ELS command when you want the command to affect the entire subsystem. For example, the ELS command display subsystem GW causes all events (except the events with 'debug' logging level) that occur through the GW subsystem to be displayed.

Event Number

Event Number is a predefined, unique, arbitrary number assigned to each message within a subsystem. In Figure 4, 019 is the event number within the GW subsystem. You can see a list of all the events within a subsystem by using the list subsystem command, where subsystem is the short name for the subsystem.

The event number always appears with a subsystem identifier, separated by a period. For example: GW.019. The subsystem and event number together identify an individual event. They are entered as a parameter to certain ELS commands. When you want a command to affect only the specified event, enter the subsystem and event number as a parameter for the ELS command.

Logging Level

Logging level is a predefined setting that classifies each message by the type of event that generated it. Use the list subsystem ELS console command to display the setting of the logging level. Table 13 lists the logging levels and types. ERROR, INFO, TRACE, STANDARD, and ALL are aggregates of other logging level types. STANDARD is the recommended default.

Table 13. Logging Levels
Logging Level Type
UI ERROR Unusual internal errors
CI ERROR Common internal errors
UE ERROR Unusual external errors
CE ERROR Common external errors
ERROR Includes all error levels above
UINFO Unusual informational comment
CINFO Common informational comment
INFO Includes all comment levels above
STANDARD Includes all error levels and all informational comment levels (default)
PTRACE Per packet trace
UTRACE Unusual operation Trace message
CTRACE Common operation Trace message
TRACE Includes all trace levels above
DEBUG Message for debugging
ALL Includes all logging levels

The logging level setting affects the operation of the following commands:

The logging level is set for a particular command when you specify it as a parameter to one of the above commands. For example:

display subsystem IP ERROR

Including the logging level on the command line modifies the display command so that whenever an event with a logging level of either UI-ERROR or CI-ERROR occurs through subsystem TKR, the console displays the resulting message.

You cannot specify the logging level for operations affecting groups or events.

Message Text

Message Text appears in short form. In Figure 4, Slf tst nt 1 int ETH/0 is the message generated by this event. Variables, such as source_address or network, are replaced with actual data when the message displays on the console.

The variable error_code is referred to by some of the Event Logging System message descriptions (usually preceded by rsn or reason). They indicate the type of packet error detected. Table 14 describes the error or packet completion codes. Packet completion codes indicate the disposition of the packets received by the device.


Table 14. Packet Completion Codes (Error Codes)
Code Meaning
0 Packet successfully queued for output
1 Random, unidentified error
2 Packet not queued for output due to flow control reasons
3 Packet not queued because network is down
4 Packet not queued to avoid looping or bad broadcast
5 Packet not queued because destination host is down (only on networks where this can be detected)

ELS displays network information as follows:

nt 1 int Eth/0 (or ) network 1, interface Eth/0,

where:

Ethernet and 802.5 hardware addresses appear as a long hexadecimal number.

IP (Internet Protocol) addresses are printed as 4 decimal bytes separated by periods, such as 18.123.0.16.

Groups

Groups are user-defined collections of events that are given a name, the group name. Like the subsystem, subsystem and event number, and logging level, use the group name as a parameter to ELS commands. However, there are no predefined group names. You must create a group before you can specify its name on the command line.

To create a group, use the add configuration command, specify the name you want to call the group, and then specify the events you want to be part of the group. The events you add to the group can be from different subsystems and have different logging levels.

After creating a group, use the group name to manipulate the events in the group as a whole. For example, to turn off display of all messages from events that have been added to a group named grouptwo, include the group name on the command line, as follows:

nodisplay group grouptwo

To delete a group, use the delete command.


Using ELS

To use ELS effectively, do the following:

When initially viewing ELS from the MONITR process, you will see a considerable amount of information. Because the device cannot buffer and display every packet under moderate to heavy loads the buffers are flushed. When this occurs the following message is displayed:

xx messages flushed

The device does not save these messages. When this message appears, tailor the ELS output to display only that information that is important to the current task you are monitoring, or use the advanced ELS commands to establish a message buffer. See Using ELS Message Buffering.

Managing ELS Message Rotation

It is also important to note that the ELS messages continually rotate through the device's buffers. To stop and restart the displaying of ELS messages, use the following key combinations:

Ctrl-S
to pause scrolling
Ctrl-Q
to resume scrolling
Ctrl-P
to go back to the last process

You may also want to capture the ELS output to a file. You can do this by starting a script file or log file from your location when Telneting to a device. You can also do this by attaching a PC to the device's console port and starting a log file from within the terminal emulation package. This information is needed to help Customer Service diagnose a problem.

Capturing ELS Output Using a Telnet Connection on a UNIX Host

Use a Telnet connection on an AIX(R) or UNIX(R) host to capture the ELS messages on your screen to a file on the host. Before beginning, set up ELS for the messages you want to capture using the ELS console commands in "Configuring and Monitoring the Event Logging System (ELS)".

To capture the ELS output to a file on an AIX or UNIX host, follow these steps:

  1. From the host, enter telnet device_ip_addr | tee local_file_name
  2. From the OPCON prompt (*), enter t 2. This accesses the MONITR process, which is the process that displays ELS messages on your screen. Depending on which ELS messages you configured, you should see ELS messages appearing on the screen.

    As long as you are in the MONITR process, all ELS messages will be written to the local file. When you exit the MONITR process (by entering Ctrl-P) or terminate the Telnet session, the logging of messages to the local file will stop.

You can also use remote logging instead of capturing ELS output on a UNIX Host. For more information about remote logging, see "Using and Configuring ELS Remote Logging".

Configuring ELS So Event Messages Are Sent In SNMP Traps

ELS can be configured so that event messages are sent to a network management workstation in an SNMP enterprise-specific trap. These traps are useful for reporting status and diagnostic results, and are often used for remote monitoring of the 2212. When ELS is configured appropriately, an SNMP trap will be generated each time the selected event occurs. For more information about SNMP, see Protocol Configuration and Monitoring Reference.

To tell ELS that a specific event should be activated to be sent as an SNMP trap, at the ELS config> prompt or at the ELS> prompt, type:

trap event ip.007
Note:If you are at the ELS config> prompt, you will need to reboot.

To enable the ELS enterprise-specific trap, follow these steps:

  1. At the SNMP config> prompt, using public as an example, type:
    SNMP config> add address public <network manager IP address>
    SNMP config> enable trap enterprise public
    SNMP config> set community access read_trap public
    

    Note:You need to reboot to activate these changes.

  2. Enable your network management station to receive and properly display the enterprise-specific traps.

Follow these steps to trap groups, subsystems, and events.


Using ELS to Troubleshoot a Problem

If you are trying to troubleshoot a particular problem, display the messages related to the problem. For example, if experiencing a problem with bridging, turn on the bridging messages:

display subsystem srt all
display subsystem br all

Initially, because of the rapid pace of messages scrolling across the screen, you may want to record the numbers you see and look them up in the Event Logging System Messages Guide manual. Once you become familiar with different types of messages being displayed for a particular protocol, you can turn on and turn off only those messages that contain the information that you require to troubleshoot a problem. The following sections list specific ELS examples. Keep in mind that different problems may require different steps.

ELS Example 1

You are interested in looking at the frequency of polling on a Token-Ring interface, and finding out whether the polls are successful.

ELS> nodisplay subsystem all all
ELS> display subsystem tkr all
Ctrl-P
* t 2

As the messages begin to scroll by, look for ELS message tkr.031.

ELS Example 2

SRB bridging is not working.

  1. Check the configuration.
  2. Use the GWCON bridging console to verify that the bridging interfaces are enabled.
  3. Enter:
    * t 6
    config> event
    ELS config> nodisplay subsystem all all
    ELS config> display subsystem srb all
    ELS config> exit
    config> Ctrl-P
    
  4. Restart the routing subsystem. When the subsystem has restarted, enter the following:
    * t 2
    

ELS Example 3

Router cannot communicate with an IPX server on an Ethernet.

  1. Enter the talk command and the PID for GWCON.
    * talk 5
    

    The console displays the GWCON prompt (+). If the prompt does not appear when you first enter GWCON, press Return.

  2. At the GWCON prompt (+), enter IPX to access the IPX console prompt (IPX>).
  3. At the IPX console prompt, enter the slist command to verify that the server is listed. (See the section on monitoring IPX in the Protocol Configuration and Monitoring Reference for information on the slist command.)
  4. Check the IPX configuration.
  5. Enter the following:
    * t 5
    + event
    ELS> nodisplay subsystem all all
    ELS> display subsystem IPX all
    ELS> display subsystem eth all
    ELS> Ctrl-P 
    * t 2
    

As the messages begin to scroll by, look for ELS message eth.001. This indicates that the server has a bad Ethernet type field.


Using and Configuring ELS Remote Logging

The remotely-logged ELS message contains all of the information that is contained in ELS messages found in the monitor queue, as viewed under talk 2, and also contains additional information as shown in Figure 5.

Figure 5. Syslog Message Description

Date/Time         IP address      Sequence Number      Local Name      ELS Subsystem Name, &
                  assigned        used for detecting   assigned        Formatted message
                  by the user     missing messages     by the user
 
Nov 20 12:13:47   5.1.1.1         Msg [0444] from      ** IBM/2212 **  :els: MPC.011 Del ent ...

Note the following differences in the remote log display:

Syslog Facility and Level

Remotely-logged ELS messages are transmitted over the network in UDP packets with the destination port number in the UDP header always equal to 514, the syslog port. To receive and process the UDP packets, the syslog daemon (syslogd) must be running in the remote workstation that is receiving and logging the ELS messages. See "Remote Workstation Configuration" for details.

Although it is not displayed in the remotely-logged ELS message, every ELS message sent on the network in a UDP packet must be assigned a syslog_facility and a syslog_level. The syslog daemon uses the combination of facility and level to determine where to route the message. Typically, you want the ELS messages to be written to one or more files in the remote host. Other options include displaying the message on the console, sending the message to one or more users, or sending the message to another workstation.

The commands you use to specify the syslog_facility and syslog_level values, along with other remote-logging related console commands, are described in "ELS Monitoring Commands" and "ELS Configuration Commands". Review these commands before reading through the next section.

Remote Workstation Configuration

The following configuration assumes that a single 2212 is remote-logging to a single remote workstation. You can configure multiple 2212s to remote-log to the same remote workstation. However, a particular 2212 can log to one and only one remote workstation. The operating system used in this example is AIX 4.2. Your environment may be slightly different. For more information on syslog, refer to the documentation for your operating system.

To perform the configuration on an AIX workstation, you must log in as root. To configure the workstation:

  1. Create or edit a syslog.conf file to specify where ELS messages with particular syslog_facility and syslog_level values are to be written. See the bottom of Figure 6 for an example of how to specify the message destination. Note that the full pathname of the log files must be specified. The default location for the syslog configuration file is /etc/syslog.conf.
  2. Create the files for logging syslog messages that you specified in the syslog.conf file.
  3. Start the syslog daemon by entering syslogd. To start the syslog daemon from SRC (System Resource Controller), enter startsrc -s syslogd. If the pathname of the configuration file is not /etc/syslog.conf, then enter syslogd -f pathname. To start the syslog daemon in debug mode, enter syslogd -d.
    Note:Running multiple instances of the syslog daemon is not supported.
  4. If the syslog daemon is already running when you create or modify the syslog.conf file, it must be restarted so that the daemon reinitializes the configuration from syslog.conf.
  5. Verify the setup by using the logger command as follows:
    logger -p user.alert  THIS IS A TEST MESSAGE (user.alert)
    logger -p news.info  THIS IS A TEST MESSAGE (news.info)
    

    If the setup is correct, THIS IS A TEST MESSAGE... will be written to the files specified in syslog.conf.

Figure 6. syslog.conf Configuration File

# @(#)34       1.9 src/bos/etc/syslog/syslog.conf, cmdnet, bos411, 9428A410j 6/13/93 14:52:39
#
# COMPONENT_NAME: (CMDNET) Network commands.
#
# FUNCTIONS:
#
# ORIGINS: 27
#
# (C) COPYRIGHT International Business Machines Corp. 1988, 1989
# All Rights Reserved
# Licensed Materials - Property of IBM
#
# US Government Users Restricted Rights - Use, duplication or
# disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
#
# /etc/syslog.conf - control output of syslogd
#
# Each line must consist of two parts:-
#
# 1) A selector to determine the message priorities to which the
#    line applies
# 2) An action.
#
# The two fields must be separated by one or more tabs or spaces.
#
# format:
#
# <msg_src_list>               <destination>
#
# where <msg_src_list> is a semicolon separated list of <facility>.<priority>
# where:
#
# <facility> is:
#      * - all (except mark)
#      kern,user,mail,daemon, auth, syslog, lpr, news, uucp, cron, authpriv, local0 - local7
#
# <priority or level> is one of (from high to low):
#      emerg,alert,crit,err(or),warn(ing),notice,info,debug
#      (meaning all messages of this priority or higher)
#
# <destination> is:
#      /filename - log to this file
#      username[,username2...] - write to user(s)
#      @hostname - send to syslogd on this machine
#      * - send to all logged in users
#
# example:
# "mail messages, at debug or higher, go to Log file. File must exist."
# "all facilities, at debug and higher, go to console"
# "all facilities, at crit or higher, go to all users"
#  mail.debug          /usr/spool/mqueue/syslog
#  *.debug             /dev/console
#  *.crit                      *
 
#   syslog messages with facilty / priority values of LOG_USER,   LOG_ALERT
user.alert            /tmp/syslog_user_alert
 
#   syslog messages with facilty / priority values of LOG_NEWS,  LOG_INFO
news.info           /tmp/syslog_news_info
 

Configuring the 2212 for Remote Logging

To configure a 2212:

  1. In talk 6, configure the remote-logging facility as shown in Figure 7. The IP address specified as the source-ip-addr should be an IP address that is configured in the 2212 for easier identification when the IP address or the hostname is shown in the remotely-logged ELS message. You should also verify that this IP address resolves quickly into a hostname by the name server or that the name server at least responds quickly with "address not found." To determine whether this happens, issue the host command on your workstation as follows:
     workstation> host 5.1.1.1
     host: address 5.1.1.1 NOT FOUND
     workstation>
    

    If the response takes more than 1 second, select an IP address which resolves more quickly.

  2. In talk 6 configure events and subsystems for remote-logging, as shown in Figure 8.
  3. Write the configuration and restart or reload the device.

Figure 7. Configuring the 2212 for Remote Logging

ELS config>set remote source-ip-addr 5.1.1.1
Source IP Addr = 5.1.1.1
 
ELS config>set remote remote-ip-addr 192.9.200.1
Remote Log IP Addr = 192.9.200.1
 
ELS config>set remote local-id ** IBM/2212 **
Remote Log Local ID = ** IBM/2212 **
 
ELS config>set remote no-msgs-in-buffer 100
Number of messages in Remote Log Buffer must be 100-512
Number of Messages in Remote Buffer = 100
 
ELS config><B>set remote facility log_news
Default Syslog Facility = LOG_NEWS
 
ELS config>set remote level log_info
Default Syslog Level = LOG_INFO
 
ELS config>set remote on
Remote Logging is ON
 
ELS config>list remote
 
------------------   Remote Log Status  -----------------
 
Remote Logging is ON
Source IP Address = 5.1.1.1
Remote Log IP Address = 192.9.200.1
Default Syslog Facility = LOG_NEWS
Default Syslog Priority Level = LOG_INFO
Number of Messages in Remote Log =  100
Remote Logging Local ID = ** IBM / 2212 **
ELS config>
 

Figure 8. Configuring Subsystems and Events for Remote Logging

ELS config>display sub snmp all
ELS config>remote sub snmp all log_news log_info
 
ELS config>display event srt.017
ELS config>remote event srt.017 log_news log_info
 
ELS config>display event stp.016
ELS config>remote event stp.016 log_user log_info
 
ELS config>display event stp.026
ELS config>remote event stp.026 log_news log_info
 
ELS config>display event stp.024
ELS config>remote event stp.024 log_news log_info
 
ELS config>display event ip.068
ELS config>remote event ip.068 log_news log_info
 
ELS config>display event ip.058
ELS config>remote event ip.058 log_news log_info
 
ELS config>display event ip.022
ELS config>remote event ip.022 log_news log_info
 
ELS config>display event gw.022
ELS config>remote event gw.22 log_news log_info
 
ELS config>display event arp.011
ELS config>remote event arp.011 log_user log_alert
 
ELS config>display event arp.002
ELS config>remote event arp.022 log_user log_alert
  
ELS config>list status
Subsystem:    SNMP
Disp levels:  ERROR INFO TRACE
Trap levels:  none
Trace levels: none
Remote levels:  ERROR INFO TRACE
        Syslog Facility/Level: LOG_NEWS LOG_INFO
 
Event     Display Trap    Trace     Remote
SRT.017   On      Unset   Unset     On
                                    Syslog Facility/Level: LOG_NEWS LOG_INFO
STP.016   On      Unset   Unset     On
                                    Syslog Facility/Level: LOG_NEWS LOG_INFO
STP.026   On      Unset   Unset     On
                                    Syslog Facility/Level: LOG_NEWS LOG_INFO
STP.024   On      Unset   Unset     On
                                    Syslog Facility/Level: LOG_NEWS LOG_INFO
IP.068    On      Unset   Unset
                                    Syslog Facility/Level: LOG_NEWS LOG_INFO
IP.058    On      Unset   Unset     On
                                    Syslog Facility/Level: LOG_NEWS LOG_INFO
IP.022    On      Unset   Unset     On
                                    Syslog Facility/Level: LOG_NEWS LOG_INFO
GW.022    On      Unset   Unset     On
                                    Syslog Facility/Level: LOG_NEWS LOG_INFO
ARP.011   On      Unset   Unset     On
                                    Syslog Facility/Level: LOG_USER LOG_ALERT
ARP.002   On      Unset   Unset     On
                                    Syslog Facility/Level: LOG_USER LOG_ALERT

Remote Logging Output

Figure 9 shows a sample from the /tmp/syslog_news_info file. Notice that the first message has a sequence number of 310. This means that the first 309 ELS messages were not sent from the source 2212. There are several reasons for this:

Notice in (1) that messages 311-313 did not get remote-logged. This is because an ARP request was outstanding and until the ARP response is received, all but the first packet is dropped in the source 2212. Additionally, the ARP cache is cleared at a user-configured refresh rate, and a new ARP request is issued. To determine when this is occurring, you can remote log events ARP.002 and ARP.011 in addition to the primary ELS events of interest. Figure 11 shows ARP events logged to the syslog_user_alert file that account for events 445 and 446, which were indicated as missing in Figure 9.

Figure 9. Sample Contents from Syslog News Info File

Nov 20 12:03:16 worksta01 root:  THIS IS A TEST MESSAGE  (news.info)
Nov 20 12:08:48 5.1.1.1 Msg [0310] from ** IBM / 2212 **: els:  IP.022: add nt 192.9.200.0 int 192.9.200.20
nt 0 int Eth/0
 
(1) ( messages 311, 312, and 313 did not get remote-logged due to ARP request outstanding - see
  explanation in the text)
 
(2) (messages 314 and 315 were logged to a separate
file - see explanation in the text)
 
Nov 20 12:08:48 5.1.1.1 Msg [0316] from ** IBM / 2212 **: els:  IP.068: routing cache cleared
Nov 20 12:08:48 5.1.1.1 Msg [0317] from ** IBM / 2212 **: els:  IP.022: add nt 5.0.0.0 int 5.1.1.1 nt 5 int Eth/4
Nov 20 12:08:48 5.1.1.1 Msg [0318] from ** IBM / 2212 **: els: SRT.017: Enabling SRT on port 5 nt 5 int Eth/4
 
 
(message 319 was logged to a separate file)
 
Nov 20 12:08:48 5.1.1.1 Msg [0320] from ** IBM / 2212 **: els:  IP.068: routing cache cleared
 
(120 messages not shown)
 
Nov 20 12:13:33 5.1.1.1 Msg [0441] from ** IBM / 2212 **: els:  GW.022: Nt fld slf tst nt 3 int Eth/3
Nov 20 12:13:33 5.1.1.1 Msg [0442] from ** IBM / 2212 **: els:  GW.022: Nt fld slf tst nt 6 int Eth/5
Nov 20 12:13:38 5.1.1.1 Msg [0443] from ** IBM / 2212 **: els:  GW.022: Nt fld slf tst nt 11 int ISDN/0
 
 
(messages 444 and 447 were logged to a separate file)
 
 
(messages 445 and 446 did not get remote-logged due to ARP request outstanding)
 
Nov 20 12:13:50 5.1.1.1 Msg [0448] from ** IBM / 2212 **: els:  GW.022: Nt fld slf tst nt 4 int PPP/0
Nov 20 12:13:50 5.1.1.1 Msg [0449] from ** IBM / 2212 **: els:  IP.068: routing cache cleared
Nov 20 12:13:50 5.1.1.1 Msg [0450] from ** IBM / 2212 **: els:  IP.058: del nt 4.0.0.0 rt via 0.0.0.4 nt 4 int PPP/0

If the initial ELS messages that are generated during and immediately after booting are of particular interest, then it is recommended that these messages also be displayed in the monitor queue, which is viewed with talk 2. Figure 10 shows the talk 2 output including the initial messages that did not get remote-logged. Note that there is a message in the talk 2 output that indicates that the remote-logging facility is available. This does not indicate that a route exists to the remote workstation, nor that the associated interface is in the "Up" state. It simply provides a reference point before which no messages can be successfully remote-logged.

Also notice that you can account for the messages that were missing (indicated in Figure 9 with (2)) in the talk 2 output.

Figure 10. Output from Talk 2

12:08:17 SNMP.024: generic trc (P2) at snmp_mg.c(766): Now 0 trap destinations
12:08:17 SNMP.012: comm public added
12:08:17 SNMP.012: comm public added
12:08:27 SNMP.022: ext err (Z1) at snmp_resconf.c(322): add_device_if_info(): sr
rdrec failed
 
12:08:27 SNMP.022: ext err (Z1) at snmp_resconf.c(322): add_device_if_info(): sr
rdrec failed
 
12:08:27 SNMP.028: err (E2) at snmp_moh.c(1583) : Duplicate
12:08:27 SNMP.028: err (E2) at snmp_moh.c(1583) : Duplicate
12:08:28   GW.022: Nt fld slf tst nt 13 int PPP/3
12:08:28   IP.022: add nt 4.0.0.0 int 4.1.1.1 nt 4 int PPP/0
 
     ( 297 messages not shown )                                             Corresponding Sequence
                                                                            Numbers in
12:08:43   GW.022: Nt fld slf tst nt 12 int PPP/2                           Remote-Logging Files :
12:08:43   GW.022: Nt fld slf tst nt 13 int PPP/3
12:08:48   IP.022: add nt 192.9.200.0 int 192.9.200.20 nt 0 int Eth/0       [0310] first message logged
12:08:48  SRT.017: Enabling SRT on port 1 nt 0 int Eth/0                    -- not logged (ARP request) --
12:08:48  STP.016: Select as root TB-1, det topol chg                       -- not logged (ARP request)--
12:08:48  STP.026: Root TB-1, strt hello tmr                                -- not logged (ARP request)--
12:08:48  ARP.002: Pkt in 1 1 800 nt 0 int Eth/0                            [0314]
12:08:48  ARP.002: Pkt in 2 1 800 nt 0 int Eth/0                            [0315]
12:08:48   IP.068: routing cache cleared                                    [0316]
 
 
     ( 126 messages not shown )
 
12:13:38   GW.022: Nt fld slf tst nt 11 int ISDN/0                          [0443]
12:13:47  ARP.011: Del ent 1 3 nt 0 int Eth/0                               [0444]
12:13:47  ARP.011: Del ent 1 3 nt 0 int Eth/0                               -- not logged (ARP request) --
12:13:47  ARP.002: Pkt in 1 1 800 nt 5 int Eth/4                            -- not logged (ARP request)--
12:13:47  ARP.002: Pkt in 2 1 800 nt 0 int Eth/0                            [0447]
12:13:50   GW.022: Nt fld slf tst nt 4 int PPP/0                            [0448]

You can use the timestamp, which appears in both the remote-logging output file and the talk 2 output, to determine when the first ELS message is successfully remote-logged. To use the timestamp for this purpose, configure ELS such that the timestamp in the monitor queue displays the time-of-day.

Also notice in Figure 9 that messages 311-313 did not get remote-logged. This is because an ARP request was outstanding and until the ARP response is received, all but the first packet is dropped in the source IBM 2212. The ARP cache is cleared at a user-configured refresh rate, and the device issues a new ARP request. To determine when ARP requests are occurring, events ARP.002 and ARP.011 can be remote-logged, in addition to the ELS events of interest. Figure 11 shows ARP events logged to the syslog_user_alert file that account for events 445 and 446, which were indicated as missing in Figure 9.

Figure 11. Sample Contents from Syslog_user_alert File

Nov 20 12:02:53 worksta01 root: THIS IS A TEST MESSAGE (user.alert)
Nov 20 12:08:48 5.1.1.1 Msg [0314] from ** IBM / 2212 **: els:  ARP.002: Pkt in 1 1 800 nt 0 int Eth/0
Nov 20 12:08:48 5.1.1.1 Msg [0315] from ** IBM / 2212 **: els:  ARP.002: Pkt in 2 1 800 nt 0 int Eth/0
Nov 20 12:08:48 5.1.1.1 Msg [0319] from ** IBM / 2212 **: els:  ARP.002: Pkt in 2 1 800 nt 0 int Eth/0
Nov 20 12:13:47 5.1.1.1 Msg [0444] from ** IBM / 2212 **: els:  ARP.011: Del ent 1 3 nt 0 int Eth/0
Nov 20 12:13:47 5.1.1.1 Msg [0447] from ** IBM / 2212 **: els:  ARP.002: Pkt in 2 1 800 nt 0 int Eth/0

You can prevent the loss of ELS messages caused by this ARP sequence by establishing a static relationship between the IP address and the MAC address. The basic steps are outlined below and are illustrated in Figure 12.

  1. In talk 5, "ping" the remote workstation's IP address
  2. In talk 5, determine the interface (net) number used to send messages to the remote-workstation's IP address
  3. Use the net number from the previous step to determine the associated MAC address
  4. In talk 6, add an ARP entry to establish a static IP address to MAC address relationship

Figure 12. Example of Setting Up a Static ARP Entry

*t 5
+p ip
 
IP>ping 192.9.200.1
PING 192.9.200.20 -> 192.9.200.1: 56 data bytes, ttl=64, every 1 sec.
56 data bytes from 192.9.200.1: icmp_seq=0. ttl=64. time=0. ms
----192.9.200.1 PING Statistics----
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0/0/0 ms
 
IP>dump
 
  Type   Dest net            Mask          Cost    Age       Next hop(s)
.
   Dir*  192.9.200.0         FFFFFF00      1       102305    Eth/0
.
IP>exit
+int
                                                    Self-Test  Self-Test   Maintenance
Net   Net'   Interface  Slot-Port                   Passed     Failed      Failed
0     0      Eth/0      Slot: 1   Port: 1                1          0           0
.
+p arp
ARP>dump
Network number to dump [0]? 0
Hardware Address        IP Address        Refresh
02-60-8C-2D-69-5D       192.9.200.1       2
 
Ctrl-P
*t 6
config>p arp
ARP config>add entry
Interface Number [0]? 0
Protocol [IP]? IP
IP Address [0.0.0.0]? 192.9.200.1
Mac Address []? 02608C2D695D
ARP config> list entry
 
Mac address translation configuration
 
IF #      Prot #  Protocol -> Mac address
   0           0  192.9.200.1 -> 02608C2D695D
ARP config>exit
Config>write
 
Ctrl-P
 
*restart
Are you sure you want to restart the gateway? (Yes or [No]): Yes
 
 (after reload, static ARP entry is active)

Additional Considerations

ELS Messages Containing IP Addresses

ELS messages containing an IP address which matches the IP address of the remote workstation will not be remote-logged, even if configured for remote-logging, and may appear under talk 2. These messages are discarded instead of being remote-logged in order to prevent excessive UDP packets from being sent on the network.

Duplicate Logging

If a facility value is repeated in syslog.conf, for example:

user.debug        /tmp/syslog_user_debug
user.alert        /tmp/syslog_user_alert

The syslog daemon will log user.debug messages only to the /tmp/syslog_user_debug file while user.alert messages will be logged to both the /tmp/syslog_user_debug file and the /tmp/syslog_user_alert file. This is consistent with the syslog design that logs the more severe conditions in multiple places.

To prevent this duplicate logging, it is recommended that different facility values be specified in the syslog.conf file. A total of 19 facility values are available.

Recurring Sequence Numbers in Syslog Output Files

Depending upon the configuration of your network, it is possible for duplicate UDP packets containing ELS messages to arrive at the remote host. It is also possible for the packets to arrive in a different order than they were transmitted. An example of this phenomenon is shown in Figure 13. Notice that the messages with sequence numbers 628 through 633 are logged twice. Also notice that after the first occurrence of sequence number 0630, sequence number 0629 occurs again, followed by the second occurrence of 0630.

Figure 13. Example of Recurring Sequence Numbers in Syslog Output

 Apr 01 10:48:33 0.0.0.0 Msg [0628] from: RA22: : els: IPX.018: SAP gen rply sent nt 5 int TKR/1, 1 pkts
 Apr 01 10:48:33 0.0.0.0 Msg [0628] from: RA22: : els: IPX.018: SAP gen rply sent nt 5 int TKR/1, 1 pkts
 Apr 01 10:49:08 0.0.0.0 Msg [0629] from: RA22: : els: IPX.037: RIP resp sent nt 0 int TKR/0, 1 pkts
 Apr 01 10:49:08 0.0.0.0 Msg [0630] from: RA22: : els: IPX.018: SAP gen rply sent nt 0 int TKR/0, 1 pkts
 Apr 01 10:49:08 0.0.0.0 Msg [0629] from: RA22: : els: IPX.037: RIP resp sent nt 0 int TKR/0, 1 pkts
 Apr 01 10:49:08 0.0.0.0 Msg [0630] from: RA22: : els: IPX.018: SAP gen rply sent nt 0 int TKR/0, 1 pkts
 Apr 01 10:49:33 0.0.0.0 Msg [0631] from: RA22: : els: IPX.037: RIP resp sent nt 5 int TKR/1, 1 pkts
 Apr 01 10:49:33 0.0.0.0 Msg [0631] from: RA22: : els: IPX.037: RIP resp sent nt 5 int TKR/1, 1 pkts
 Apr 01 10:49:33 0.0.0.0 Msg [0632] from: RA22: : els: IPX.018: SAP gen rply sent nt 5 int TKR/1, 1 pkts
 Apr 01 10:49:33 0.0.0.0 Msg [0632] from: RA22: : els: IPX.018: SAP gen rply sent nt 5 int TKR/1, 1 pkts
 Apr 01 10:50:08 0.0.0.0 Msg [0633] from: RA22: : els: IPX.037: RIP resp sent nt 0 int TKR/0, 1 pkts
 Apr 01 10:50:08 0.0.0.0 Msg [0633] from: RA22: : els: IPX.037: RIP resp sent nt 0 int TKR/0, 1 pkts

Because neither Syslog nor UDP has the ability to handle duplicate or out of sequence packets, it is important to recognize the possibility of duplicate sequence numbers occurring.


Using ELS Message Buffering

Message buffering is an advanced feature of ELS that can help you with problem determination. You can set up defaults that ELS will use for message buffering or change how messages are buffered while the device is operating. Message buffering can minimize the information lost because messages have wrapped in the default message buffers. Message buffering is accessible through the advanced configuration or monitoring command. It enables you to:

For specifics about the commands, see ELS Message Buffering Configuration Commands and ELS Message Buffering Monitoring Commands.

The following example shows how to configure ELS message buffering.
Note:Setting of the Advanced ELS buffer size must be performed under talk 6. The remaining setup steps can be performed under either talk 5 or talk 6.

*t 6
Gateway user configuration
Config>event
Event Logging System user configuration
ELS config>advanced
Advanced ELS Config Console
ELS Config Advanced>set buffer
Enter buffer size in range 0 to 15219 KB [5073]?
Buffer size set to 5073 KB
NOTE: Any more config changes made before rebooting
could affect the availability of sufficient memory after
reboot!
ELS Config Advanced>exit
ELS config>exit
Config>write
Config>
*reload
Are you sure you want to reload the gateway? (Yes or [No]): Yes
 
 
(after reloading...)
 
 
*t 5
 
CGW Operator Console
 
+event
Event Logging System user console
ELS>advanced
Advanced ELS Console
ELS Advanced>list status
------------------Advanced ELS Configuration------------------------
Logging Status:  OFF  Wrap Mode:  ON   Logging Buffer Size: 5073 KB
Stop-Event:  NONE   Stop-String:   NONE
Additional Stop-Action:  NONE
------------------------Run-Time Status-----------------------------
Has Stop Condition Occurred?   NO   Messages currently in buffer:  0
 
ELS Advanced>set stop event gw.26
Stop Event "GW.026" has been set
ELS Advanced>set stop string Mnt nt 5
Stop String set to "Mnt nt 5"
ELS Advanced>set stop action APPN-DUMP
Stop Action has been set to APPN-DUMP
ELS Advanced>set wrap off
Advanced Wrap Mode set to OFF.
 
ELS Advanced>log subsys gw all
ELS Advanced>set logging on
Advanced Logging set to ON.
ELS Advanced>list status
------------------Advanced ELS Configuration------------------------
Logging Status:  OFF  Wrap Mode: OFF  Logging Buffer Size: 5073 KB
Stop-Event:     GW.026   Stop-String:  Mnt nt 5
Additional Stop-Action:  APPN-DUMP
------------------------Run-Time Status-----------------------------
Has Stop Condition Occurred?   YES  Messages currently in buffer:  2
 
ELS Advanced>view all noscroll
 
[1] 10:52:10   GW.026: Mnt nt 0 int Eth/0
[2] 10:52:10   GW.026: Mnt nt 5 int Eth/1 (1)
 

(1) This triggered the stop action.


[ Top of Page | Previous Page | Next Page | Table of Contents | Index ]